Tanzu Kubernetes Grid nodes are unable to resolve hostnames with the ".local" domain suffix

Products

• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

Symptoms:
Attempts to resolve hostnames that have a domain suffix that ends in ".local" fail on Tanzu Kubernetes Grid (TKG) nodes.

Environment

VMware Tanzu Kubernetes Grid 1.x
VMware vSphere 7.0 with Tanzu
VMware Tanzu Kubernetes Grid Plus 1.x

Cause

This is not unique to TKG. This is expected behavior from the systemd-resolved service. See https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html for more detailed information on this topic.
 

• Multi-label names with the domain suffix ".local" are resolved using MulticastDNS on all local interfaces where MulticastDNS is enabled. As with LLMNR, IPv4 address lookups are sent via IPv4 and IPv6 address lookups are sent via IPv6.

• Queries for multi-label names are routed via unicast DNS on local interfaces that have a DNS server configured, plus the globally configured DNS servers if there are any. Which interfaces are used is determined by the routing logic based on search and route-only domains, described below. Note that by default, lookups for domains with the ".local" suffix are not routed to DNS servers, unless the domain is specified explicitly as routing or search domain for the DNS server and interface. This means that on networks where the ".local" domain is defined in a site-specific DNS server, explicit search or routing domains need to be configured to make lookups work within this DNS domain. Note that these days, it's generally recommended to avoid defining ".local" in a DNS server, as RFC6762 reserves this domain for exclusive MulticastDNS use.

Resolution

This is a know issue affecting Tanzu Kubernetes Grid. There is currently no resolution.

Workaround:
To workaround this issue you will need to deploy your management and workload clusters with a modified plan that will update the name resolution parameters on the TKG nodes as they are deployed.
 

Nameservers on vSphere provides the instructions needed for using custom DNS servers on your TKG nodes. These instructions only need one modification to allow for domain names ending in ".local" to be resolved. A "searchDomains" line needs to be added to the end of the vsphere-overlay-dns-control-plane.yaml and vsphere-overlay-dns-workers.yaml files. Once modified, the end of these files should look like the following:

nameservers: ["x.x.x.x"]
searchDomains: ["xxxxxxxx.local"]

Note: Replace "x.x.x.x" with your DNS server and "xxxxxxxx.local" with your domain suffix.

Additional Information

vSphere with Tanzu unable to resolve hostnames with the ".local" suffix (83387)

Impact/Risks:
The .local hostname is reserved for use in mDNS per RFC6762  therefore trying to resolve it against a DNS server violates RFC6762. As such VMware does not recommend any deployment which uses .local for any components. (this includes vCenter, ESXi, NSX Advance Load Balancer, NSX manager, NSX Edge nodes, TKG nodes or API endpoints, and any endpoint TKG uses like harbor).

The workaround for this is strictly for proof of concept and lab use. Implementing this workaround in your production environment could result in unexpected scenarios.